feat: implement schema comparison logic #50400
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
octavia-squidington-iii
added
area/connectors
|
/format-fix
|
...-v2/src/main/kotlin/io/airbyte/integrations/destination/iceberg/v2/IcebergTypesComparator.kt
Show resolved
Hide resolved
| * - Columns that no longer exist in the incoming schema (i.e., removed). | ||
| * - Columns that changed from required to optional. | ||
| */ | ||
| class IcebergTypesComparator { |
There was a problem hiding this comment.
wondering: which of these is better
- diff existing vs incoming iceberg schema (your current pr)
- diff existing iceberg schema vs incoming AirbyteType (probably dumb)
- convert existing iceberg schema to AirbyteType; diff existing vs incoming AirbyteType (ie. sharable diff logic)
I think diffing iceberg schemas (i.e. keep your current approach) is better? b/c (a) iceberg types don't perfectly align with airbyte types, and (b) iceberg is weird b/c it has recursive schemas?
There was a problem hiding this comment.
Yup! I did try to make it generic but realised it's not worth trying to solve a future problem when we dont know enough about it. I think for now this being iceberg specific is fair enough and we can re-visit it once we have more understandings of how this can be generalised and made better
Part of https://github.com/airbytehq/airbyte-internal-issues/issues/10918